Metric entropy limits on recurrent neural network learning of linear dynamical systems

One of the most influential results in neural network theory is universal approximation theorem [1], [2], [3] which states that continuous functions can be approximated to within arbitrary accuracy by single-hidden-layer feedforward networks. The purpose this paper establish a result spirit for general discrete-time linear dynamical systems—including time-varying systems—by recurrent networks (RNNs). For subclass time-invariant (LTI) systems, we devise quantitative version statement. Specifically, measuring complexity considered class LTI systems through metric entropy according [4], show RNNs optimally learn—or identify system-theory parlance—stable systems. whose input-output relation characterized difference equation, means learn equation from traces metric-entropy optimal manner.